Aspiration-driven co-evolution of cooperation with individual behavioral diversity

In evolutionary game, aspiration-driven updates and imitation updates are the two dominant game models, and individual behavior patterns are mainly categorized into two types: node player and link player. In more recent studies, the mixture strategy of different types of players has been proven to improve cooperation substantially. Motivated by such a co-evolution mechanism, we combine aspiration dynamics with individual behavioral diversity, where self-assessed aspirations are used to update imitation strategies. In this study, the node players and the link players are capable to transform into each other autonomously, which introduces new features to cooperation in a diverse population as well. In addition, by driving all the players to form specific behavior patterns, the proposed mechanism achieves a survival environment optimization of the cooperators. As expected, the interaction between node players and link players allows the cooperator to avoid the invasion of the defector. Based on the experimental evaluation, the proposed work has demonstrated that the co-evolution mechanism has facilitated the emergence of cooperation by featuring mutual transformation between different players. We hope to inspire a new way of thinking for a promising solution to social dilemmas.


Introduction
Cooperative behavior commonly exists in nature and human society [1].From the perspective of the whole environment, spontaneous cooperative behavior is conducive to the sustainable development of the group and the maximization of the overall interests of the group [2].But for individuals, defectors usually gain more than cooperators due to competitive relationships and limited resources, which means that selfish defection often results in a greater payoff.In the environment of fittest survival, the defection strategy seems to be an optimized solution according to the law of survival [3][4][5][6].With the emergence of the contradiction between the individual's defection and the group's cooperation, as a consequence, why cooperative behavior emerges and is maintained in selfish individuals has become a widely concerned topic.
To solve the contradiction and facilitate the emergence of cooperation, different approaches based on evolutionary game theory [7,8] have been proposed.Nowak effectively combined network reciprocity with evolutionary game theory in the ways of kin selection [9,10], direct reciprocity [11][12][13], indirect reciprocity [13,14], network reciprocity [15][16][17], and group selection [18].By investigating the evolution of cooperation on complex networks, it was found that different network topologies have a significant effect on the generation of cooperative clusters [19][20][21][22][23][24].On the other hand, the aspiration-driven dynamic [25][26][27] and incentive direction [28][29][30][31] are also effective methods of promoting cooperation.In imitation-driven dynamics, individuals adjust their strategies based on the gains obtained from the game compared to their aspirations.In contrast, individuals imitate the strategies of more successful peers in aspiration dynamics [32].
However, most current studies on games are based on a strict node-to-node pattern in which each node player plays with a uniform strategy towards its surrounding neighbors [33][34][35].In this pattern, some limitation is still hard to be avoided, for example, a player adopts a defection strategy to defend against a defector's invasion in the neighborhood, meanwhile, this behavior may harm nearby cooperators as a side effect.Based on the interactive heterogeneity of players [36], the "link players" are introduced into the game to adopt different strategies when playing games with different neighbors [37,38].
Studies on link player games have shown that the node players and the link players have different properties in different network topologies [36,39].In structured networks, link players are able to reach higher levels of cooperation in the Prisoner's Dilemma (PD) game [40].Inspired by these findings, Su et al. argued that the strategies of link players evolve in the form of behavior diffusion along edges in the game [41,42].The discrepancy in behavioral patterns of the node player and link player leads to different evolutionary outcomes.A study by Jia et al. revealed the effect of a mixed crowd community of link players and node players on cooperation and also proved that mixed groups of link and node players enable a higher level of cooperation.When the density of link players is low, node players consciously form cooperative groups to resist defection [43].
More recently, researchers have been exploring mixed-player games involving both node players and link players, aiming to better understand co-evolution mechanisms that drive the promotion of cooperation.However, many of these studies have taken a somewhat uniform and statistical approach [43].Drawing from insights in sociology, it's been recognized that individual mindsets, ranging from radicalism to conservatism [44,45], can significantly influence decision-making.In this context, conservatism refers to the inclination to maintain the existing status quo, whereas radicalism involves challenging or altering the status quo.
To address these considerations, we've introduced a novel perspective in our work.We've embraced the interconnected nature of radicalism and conservatism and combined them with aspiration-driven dynamics to create a dynamic aspiration mechanism.This innovative approach allows for a continuous exchange between node players and link players, enabling them to transition between these roles organically.Our experimental results underscore the efficacy of this mechanism, demonstrating a marked improvement in cooperation levels.Furthermore, this mechanism enhances the system's ability to adapt to changing environments.

Materials and methods
In the proposed model, the study of repeated PD games is conducted on an L×L periodic lattice network [46].The payoff of the PD is usually represented by a two-dimensional matrix as: Suppose that two participants simultaneously decide to take cooperation C or defection D, the payoffs caused by different actions contain reward R, punishment P, temptation T, and a sucker's payoff S. For instance, to obtain payoff S, the strategy from the first row C needs to meet the strategy from the second column D. The PD requires the conditions T>R>P>S, 2R>S+T [47].The values of R, S, T, and P are set to 1, 0, b, 0, respectively.Specifically, b is set between 1 and 2 by following a game set-up criteria called the "weak Prisoner's Dilemma" [48].
The repeated PD game is simulated in a periodic lattice network, in which each node represents a player and the connections between them are the relationship.The player plays a PD with its surrounding neighbors by setting L to 100.There are two types of players in the network: node player and link player.Node players adopt a uniform strategy towards their neighboring nodes, while link players take different strategies towards different neighbors.During initialization, all players are initialized as node players, who have an equal probability to adopt a cooperation or defection strategy.
In the game, player x is randomly selected from all participants, then the selected player x randomly chooses a surrounding neighbor y to play to gain P xy .The payoff P x of player x is defined as the sum of the gains of the game between player x and all surrounding neighbors, as follows: where O x is the set of four neighbors around player x.Players have the dynamic aspiration property, and the term 'aspiration' here denotes a player's expectation of the payoff in the game.Noted that the aspiration evolves as the game evolves, the aspiration of player x is only related to the payoff of its own and the previous aspiration.Furthermore, aspiration can also act as an indicator of the player's satisfaction.If player x's aspiration is greater than or equal to that in the previous round, the state of x can be regarded as satisfied, otherwise is unsatisfied.
The equation of aspiration is: where P x (t) denotes the payoff of player x in round t, A x (t) denotes player x's aspiration in round t, and a (0<a�1) indicates the player's learning rate of aspiration (0.1 by default).Player x's aspiration in round t+1 depends on the player x's payoff and aspiration in round t.According to Formula (3), when the game reaches equilibrium, the type of player won't change.So, A x (t+1) should be equal to A x (t), which means P x (t)−A x (t) should be zero.In other words, when the game reaches equilibrium, player x's payoff should be equal to player x's aspiration.That's why player x's aspiration converges to p on the condition that the payoff of player x is a stable value p.If all players adopt a cooperative strategy in the game sustainably, players will get the maximum payoff of 4. Then the value domain of aspiration is (0,4].The aspiration of all players is initialized to 2, i.e.A x (0) = 2. Using aspiration, we build a set of mechanisms for node and link players to transform each other cyclically: where N x denotes that player x is a node player and L x denotes that player x is a link player.The mechanism of co-evolution of node players and link players is triggered after player x updates its strategy.In case of adversity, individuals tend to become radical and try to change the current situation.Conservative behavior is a better choice for individuals to maintain the status quo in good times.When node player x is dissatisfied at the time A x (t+1)<A x (t), node player x will be in a state of radicalism and transform into a link player.When link player x is satisfied at the time A x (t+1)�A x (t), link player x will be in a state of conservatism and transform into a node player.
In the process of interconversion between node players and link players, it is necessary to pay attention to some differences in the traits of node and link players.When player x is transformed from a node player to a link player, the strategy of handling the surrounding neighbors is uniformly changed to the original strategy.When player x is converted from a link player to a node player, it cannot be directly converted to a node player because of the heterogeneity of the link player's strategy.To solve this problem, the proportion of the four edges that adopt cooperation and defection strategy is calculated respectively.This proportion is used as the probability of adopting a cooperation (defection) strategy after the link player transforms into a node player.For example, suppose that the link player x has four neighbors y 1 , y 2 , y 3 , and y 4 , and it adopts a cooperative strategy in the game with neighbors y 1 , y 2 , and y 3 , and a defection strategy in the game with the neighbor y 4 .After the link player x is transformed into a node player, the probability of cooperative strategy adaptation is C CþD ¼ 0:75 and the probability of defection strategy adaptation is D CþD ¼ 0:25.The detailed transformation mechanism is illustrated as Fig 1: When the player x selects a neighbor for a PD game, the classic Fermi function is used to update the strategy of x [49].
The Formula (5) represents the probability that player x imitates player y 0 s strategy after playing with player y, where P x and P y respectively denote the payoff of players x and y, and S x and S y denote the strategies of players x and y.K is the random noise describing the uncertainty of the strategy update.Here, when imitation actions occur, four situations will occur.(a)If both x and y are node players, x will simply imitate the strategy of y. (b)If x is a node player and y is a link player, x will imitate the strategy on the link between x and y. (c)If both x and y are link players, x will imitate the strategy on the link between x and y. (d)If x is a link player and y is a node player, the strategy on the link between x and y will imitate the strategy of y.To simplify the experiment without losing generality, the K is set to 0.1.The Monte Carlo (MC) steps are set to 5000.To ensure correct accuracy, each set of parameters was repeated 20 times independently and the final results are averaged.

Results and analysis
To verify the effectiveness of the proposed co-evolution mechanism of node and link players, substantial simulations are conducted on the overall frequency of cooperation (ρ c ) in the game.
Fig 2 shows the relationship between the overall frequency of cooperation (defection) and the percentage of node (link) players, both of them are influenced by the value of b.The overall frequency of cooperation (ρ c ) under the co-evolution mechanism of node and link players is maintained at a high level compared to the traditional PD game.As we can see, as the value of b increases, the overall frequency of cooperation (ρ c ) and the proportion of node players decrease together, but the overall frequency of defection and the proportion of link players increase together.When 1.1<b<1.6,ρ c decreases quickly and soon remains at a certain percentage as the game becomes more difficult.If b>1.6, ρ c slowly decreases to maintain a relatively stable state, and the co-evolution mechanism allows cooperative players to survive.The main players are composed of defectors, and cooperative players account for a relatively small percentage.
From Fig 2, it can be observed that there is a strong similarity between the trend of the percentage of node players and the trend of the cooperation frequency (ρ c ).By the same token, the percentage of link players and the frequency of defection also reveal this phenomenon.This demonstrates that there is a relationship between the percentage of both types of players and the overall frequency of cooperation.
To further study the relationship between the percentage of both types of players and the frequency of cooperation (ρ c ), we also investigate the evolution of cooperative strategy among two types of players, in which the value of b is set to 1.2.As shown in Fig 3, the proportion of link players is rapidly increasing at the beginning because of the transformation from node players to link players.The main reason is that most node players are at a disadvantaged point in the game because of the weak survivability of node players, only a small proportion of node players survive and gradually form cooperative clusters.In the subsequent evolution, the link players around cooperative groups (composed of node players) are gradually transformed into node players, which is the result of the aspiration of link players keeping rising.The cooperation frequency of link players (ρ lc ) is lower than that of node players (ρ nc ), as a consequence, the link players play more of a role in promoting cooperation.Eventually, both types of players reach the equilibrium state, and the conversion between node players and link players no longer takes place.
To present more effect details of the evolutionary game process, snapshots of the characteristics are taken from a microscopic perspective at different b values in Fig 4 .It can be easily found that link players are not present at the beginning and then rapidly appear abundantly.At b = 1.04, node players disappear firstly and then reoccupy the body of the network in the subsequent evolution.But at b = 1.2, link players always occupy the body of the network.In the process of evolutionary game, a minority of node players with cooperation strategy gather into groups and form cooperative communities.Node players with defection strategy and link players with defection strategy gather at the edge of the group of node players with cooperation strategy.Eventually, all node players with defection strategy become extinct, the mutual transformation of node players and link players tends to balance.The link players surround the node players to form an isolation zone to ensure the survival of node players with a cooperation strategy.The game mechanism between a group of node players with cooperative strategy and players at the edge of the group is significantly different from the traditional game model, which promotes unique behavioral patterns for both types of players.
The co-evolution mechanism drives node and link players to form specific behavior patterns, as shown in Fig 5 .The role played by the co-evolution mechanism can be summarized in two aspects: on the one hand, to prevent the demise of the cooperator community (Fig 5 (A)), and on the other hand, to promote the growth of the cooperator community (Fig 5 (B)).In the process of the game, the surviving node players generally adopt a cooperation strategy.As we can see in Fig 5(A), node players are transformed into link players to obtain part of the benefits of cooperation strategy, to defend against players adopting defection strategy.The node player adopting the defection strategy loses the exploitative target and transforms into the link player because of the extinction of the node player adopting the cooperation strategy.As shown in Fig 5(B), node players adopting the defection strategy occupy the edge of the group which is composed of node players adopting the cooperation strategy, and they often do not survive for long.Some of them eventually transform into node players adopting the cooperation strategy, while others transform into link players, depending on the gain between cooperation strategy and defection strategy.There are wide variations in the frequency of cooperation at different game difficulties, but the co-evolution mechanism performs a very similar function.The co-evolution mechanism serves to protect node players with cooperation strategy, which is an important reason why node players are surrounded by link players adopting cooperation strategy when the game is in equilibrium.With the mechanism, node players with cooperation strategy and defection strategy, and link players can be transformed into The mutual transformation of different types of players is closely related to aspiration, and snapshots of the aspiration are captured in the evolution.As shown in Fig 6, it is obvious that the evolution of player aspiration is surprisingly consistent with the evolution of cooperation.The player's aspiration reflects the player's payoff.When evolution is in the state of equilibrium, the player's aspiration is as same as the player's payoff.The group clusters are formed by node players via adopting cooperation strategy, and the corresponding gain is stable at 4 in evolutionary equilibrium, which means that the aspiration of the node players in the group finally converges to 4. For the link players surrounding the node players, they have at least one With different initial values, the effect of aspiration on the frequency of cooperation is explored, and the results are shown in Fig 7 .In the previous experiment, the aspiration is initialized to 2 (ranging in (0,4]), which may ensure the universality of the experimental results to the maximum extent.When b<1.1, the frequency of cooperation reaches a very high level regardless of the initial value of aspiration, and when b>1.1, the frequency of cooperation decreases rapidly.As the initial value of aspiration increases, the survival ability of the cooperator is improved, cooperator can maintain a certain frequency of cooperation in a hard PD environment.In addition, it can be seen that the frequency of cooperation reaches its maximum when the initial value of aspiration is set to 2.

Conclusion
In this paper, an evolutionary game system is established, where the node players and link players can be transformed into each other, to study the evolution of cooperation.Compared to the static model proposed by Jia et al., the model proposed in our study is dynamic.In our study, the node players and link players can be transformed into each other through aspirations.We introduce the aspiration inspired by the mindsets between radicalism and conservatism.Here, the aspiration acts as an indicator of the player's satisfaction, which will evolve as the game evolves.After performing numerical simulations, we find that the level of cooperation can be effectively promoted under the proposed mechanism.The co-evolution mechanism drives node and link players to prevent the demise of the cooperator community by forming specific behavior patterns.With an isolation zone, the node players are surrounded by the link players to ensure the survival of cooperator.Moreover, there is a striking consistency in the evolution of player aspiration and cooperation.The combination of aspirationdriven dynamics and individual behavioral diversity has a particular contribution to the evolution of cooperation.The co-evolution mechanism is more in line with the characteristics of the species diversity in nature, and more closely resembles the natural evolution.We hope to attract increasing attention and interest to continue along this line of study as well.

Fig 1 .
Fig 1. Panels (a)-(b) show the specific process of interconversion between node players and link players.(a) shows the process of converting a node player to a link player, the link player treats its surrounding neighbors uniformly with the original strategy.(b)shows the transformation of a link player to a node player.Since node player can only adopt one strategy, link player is converted into node player who adopts cooperation (defection) strategies based on the percentage of cooperation (defection) strategies.https://doi.org/10.1371/journal.pone.0291134.g001

Fig 2 .
Fig 2. The trend of the overall frequency of cooperation and defection at different b values is related to the percentage of both types of players for a = 0.1.https://doi.org/10.1371/journal.pone.0291134.g002

Fig 3 .
Fig 3. Comparison of the proportion of two types of players, the frequency of cooperation of two types of players, and the overall frequency of cooperation.All results are shown for b = 1.2.https://doi.org/10.1371/journal.pone.0291134.g003

Fig 4 .
Fig 4. The node players with cooperation strategy survive because of the isolation zone formed by being surrounded by link players.Red, blue, gold, and cyan denote node players with cooperation strategy, node players with defection strategy, link players with cooperation strategy, and link players with defection strategy, respectively.The two rows of snapshots correspond to b values of 1.04 and 1.2, respectively.The images from left to right show the number of MC steps as 0, 5, 100, and 3000 respectively.https://doi.org/10.1371/journal.pone.0291134.g004

Fig 5 .Fig 6 .
Fig 5. Patterns of behavior among node and link players under the co-evolution mechanism play a role in facilitating cooperation.(a) shows the behavior pattern of the node players adopting a cooperation strategy.(b)shows the behavior pattern of node players adopting the defection strategy.Node players with cooperation strategy, node players with defection strategy, and link players transform into each other as shown in (c).https://doi.org/10.1371/journal.pone.0291134.g005

Fig 7 .
Fig 7. The effect of different initial values of aspiration on the frequency of cooperation.When the aspiration is initialized to 2, the frequency of cooperation reaches an extreme value.https://doi.org/10.1371/journal.pone.0291134.g007